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REMARKS 

Information Disclosure Statement 

. The Examiner requeste.d..copies.of some references. mentioned In. the 

specification. In response, the applicants enclose the requested references. 

The Objection to the Drawinaa 

The drawings were objected to because the description of FIGs. 3D and 3E 
on page 14, lines 1 8-20 calls for red curves and the drawings are In black In white. 
In response, the applicants have amended the specification to clarify which cun/es 
are being refen-ed to. Acceptance of this change Is courteously requested. 

The Section 37 CFR 1.75 Objection of Claims 1-12 and 21-28. 

Claims 1-12 and 21-28 are objected to under 37 CFR 1.75, as being 
indefinite. The applicants have amended Claims 1 and 1 1 to provide proper 
antecedent basis. It is believed that the foregoing amendments to Claims 1 and 
1 1 have clarified any indeflniteness that existed in the original claim language. 

It is believed the amended claims now fulfill the requirements of 37 CFR 
1.75, as they particularly point out and distinctly claim the subject matter which 
the applicant regards as the invention. Therefore, it is respectfully requested that 
the objection to Claims 1-12 and 21-28 be reconsidered based on the amended 
claim language. 

The 35 use 102 Rejection of 1-3. 6. 7 and 12. 

Claims 1-3, 6, 7 and 12 were rejected under 35 USC 102(b) as being 
anticipated by Wang et al„ U.S. Patent No. 5,557,684 A, herein after refen-ed to as 
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Wang. It was contended in the aljove-identified Office Action that Wang teaches all 
the elements of the rejected clainris. 

While no admission is made that these claims are actually anticipated by the 
cited reference, the applicants have chosen to incorporate the subject matter of 
Claim 8 Into Claim 1 to further the prosecution of the application and expedite Its 
allowance. Claim 8, was stated in the Office Action as being allowable if rewritten in 
Independent fonn including all limitations of the base claim and any intervening 
claims. This is essentially what has been done by incorporating the subject matter 
of Claim 8 into Claim 1. In addition, Claim 9 was made dependent from Claim 1, to 
confonn It to the foregoing change. Since the incorporation of the subject matter of 
Claim 8 into Claim 1 now makes Claim 1 allowable, all the rejected claims that have 
not been cancelled, Claims 1-7, 9-12 and 21-28 which ultimately depend from 
Claim 1 , are allowable as well. Claim 8 has been cancelled. 

The 35 use 103 RejBCtion of Claims 5 and 10. 

Claims 5 9nd 10 were rejected under 36 USC 103(a) as unpatentable over 
Wang et al., U.S. Patent No. 5,557,684 A), In view of Kaup et al. 

While no admission is made that these claims are actually made obvious by 
the cited references, the applicants have chosen to incorporate the subject matter of 
Claim 8 Into Claim 1 to further the prosecution of the application and expedite its 
allowance, as discussed above. Claims 5 and 10 were made dependent from Claim 
1 , to conform it to the foregoing change. Since the incorporation of the subject 
matter of Claim 8 into Claim 1 now makes Claim 1 allowable, all the rejected claims 
that have not been cancelled, Claims 1-7, 9-12 and 21-28 which ultimately depend 
from Claim 1, are allowable as well. 
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Ordinal characteristics of transparency. 
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Figure 1 shows an example of visual transparency. The Image could arise from 
a number of different physical causes. For example, a square of tissue paper 
could be in front of a dark grey circle; or a circular shadow could be cast on a 
plane containing a light grey square; or a dark circular filter could be lying on 
top of a light grey square. Although the physics is uncertain, one can perceive 
the image as a combination of two more primitive Images. 




Figure 



We use the term "transparency" to cover the general case of such Image 
combination, Including what would be called "translucency" in ordinary 
language. Many physical phenomena can produce transparency. For 
example, dark filters, specular reflections, puffs of smoke, gauze curtains, and . 
cast shadows, all combine with patterns behind them in a transparent manner. 
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When an image has been formed by the combination of two primitive images, 
then it is usually more parclmonious to describe the image in terms of that 
combination; thus it is advantageous for a visual system to parse the image into 
the primitive images along with a combination rule. This parsimony does not 
depend on assigning a unique physical interpretation to the primitive images; 
figure 1 can be parsed into a circle and a square, even in the absence of a 
decision about the underlying physics. 

We suggest that visual transparency may be initially analysed at a "pre- 
physical** level, which does not include the physical specificity of a full intrinsic 
image analysis [1]. The representation at this level consists of a set of primitive 
image layers which are ordered In depth. Each layer contains filled regions 
which modify the appearance of the layers beneath them, and unfilled regions 
which are perfectly clear The filled regions of different layers combine with 
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each other according to simple rules such as multiplication and addition. Figure 
2 shows an example of the layers which might give rise to figure 1 In the 
simplest layered model luminance only propogates from the lower layers to the 
higher ones (i.e. toward the viewer). 



ill 
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Figure 2 



The significance of X junction?^ 



When patterns on distinct layers overlap, they typically give rise to X junctions In 
the image, which have an important influence on the perception of transparency 
by humans [2 - 4]. These X junctions can be quite diagnostic of the nature of the 
transparent interaction, and the depth ordering of layers. For example, figures 
3(a-c) contain three Images, which the human visual system Interprets in three 
different ways. Transparency can be seen In figure 3(a), which Is Interpreted as 
containing two dark filters; however, the depth ordering is ambiguous: either 
square can be seen as lying in front of the other. Transparency can also be 
seen in figure 3(b), but In this case the depth ordering is unambiguous: the 
square on the lower-ieft appears to be in front Finally, transparency is not seen 
in figure 3(c), which is commonly seen as a painted pattern lying In a single 
layer. 
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Figures 

The qualitative characteristics of the transparency can be related to the 
qualitative characterlsllcs of the X junctions. Let p^q^ns be the luminances In the 
four regions sun^ounding the X junction, as indicated In figure 3(d), In figure 3(a) 
p<<7 and /<s, which is to say that the vertical edge retains the same sign In both 
halves of the X junction. Similarly, p<r and q<$, which is to say that horizontal 
edge, also retains the same sign in both halves of the X junction. We call this a 
"non-reversing" junction because both edges retain their sign. In figure 3(b) the 
vertical edge changes sign across the X junction, whereas the horizontal edge 
retains its sign. We call this a "single-reversing'* junction. Finaliy, in figure 3(c) 
both the horizontal and the vertical edges change sign across the X junction. 
We call this a "double-reversing" junction. 

The human visual system seems to employ heuristics related to these different 
categories of X junctions. Non-reversing junctions support the perception of 
transparency, whii© leaving the depth ordering of the layers ambiguous. Single- 
reversing junctions also support transparency, and in addition Impose a unique 
depth ordering. Double-reversing junctions do not support transparency. The 
junctions thus offer pieces of local evidence which may be propogated through 
the figure to the interpretation of transparency and depth order. 

Computational analysis . 

We may examine transparency from a computational point of view, to 
understand the basis for the heuristics described above. We begin with a 
framework to characterize the combination of transparent layers; the layers will 
be denoted h - In- Each layer may attenuate the luminance from the layer 
beneath it by a factor a, 0 < a S 1 , and may contribute Its own emission of 
quantity e, e 20. The attenuation and emission are functions of position, 
and e(x,>). An unfilled region has a = 1 and e - 0. (This formulation is slightly 
different from Metelli's, but the resulting restrictions are similar to those derived 
from iVleteiii's rules). 

If layer n-1 contains a luminance pattern /n.i(x,y), then the luminance pattern at 
layer n is: 

/n(xy) = an(x,>^. /fv^(x,y) en{x,i^ (1 ) 
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Since both multrplicative and additive Interactions are allowed, a wide range of 
(p,qj,s) values are legal examples of transparency. However, there are 
constraints on those values. The allowed ranges of a and e Imply that a filled 
region must reduce or leave unchanged the amplitude of the luminance 
variation in a lower layer This allows us to establish some Inequalities 
concerning the X Junction of figure 3(d). We assume that an X-junction results 
from the overlap of filled regions in two layers; it remains to determine whether 
the frontal layei^s edge is vertical or horizontal, and which half of the edge rs 
filled 

The four possible local hypotheses about the filled frontal region are: (I) It lies 
above the horizontal line, (li) it lies below the horizontal line, (III) It lies to the left 
of the vertical line, and (iv) it lies to the right of the vertical line. The conditions 
on the attenuation factor translate into the following inequality conditions. 

(1) hypothesis (i) Is physically plausible iff 0< (p-Q)/(r-s) 5 1, 

(2) hypothesis (ii) is plausible iff 0< (r-s)/{p-(7) 

(3) hypothesis (iii) is plausible Hf 0 < (p-r)/(fl-s) 5 1 , and 

(4) hypothesis (iv) is plausible iff 0 < {q-s!)/{p't) 5 1. 

Note that conditions (i) and (ii) are mutually exclusive unless the ratio is unity, 
likewise for conditions (iii) and (iv). 

The fact that these ratios are non-negative leads to the edge-reversal heuristics 
noted above. Thus an edge which is tranparently occluded cannot reverse 
sign, while an edge which Is in front may or may not reverse sign. It follows that 
double-reversing junctions have two consistent interpretations, and single- 
reversing junctions have only one. A double-reversing junction would require 
that both the vertical and horizontal edges be in front of the other, which Is 
impossible; therefore no transparent Interpretation is allowed. 



Conclusion 

Transparency can arise in images due fo a number of different physical 
phenomena. We have proposed a pre-physical level of representation in which 
a number primitive images organized as [ayers combine together to fonn an 
observed image. The ordinal relationships between the luminances at an X 
junction can be used categorize the X junction as non-,singIe-»and double- 
reversing junctions. These categories can be determined without precise 
measurements, and are robust against point nonlinearities in luminance 
sensitivity. Non- and single-reversing junctions support transparency; single- 
reversing junctions lead to an unambiguous interpretation of depth-order of the 
layers^ while non-reversing junctions leave the depth-order ambiguous. 
Double-reversing junctions do not support transparency. Propogation of these 
constraints can be used to rapidly restrict the set of the legal interpretations of 
an image. 
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Recovering High Dynamic Range Radiance Maps from Photographs 

Paul E. Debevec Jitendra Malik 

University of California at Bericdey^ 



ABSTRACT 

Wc present a tn^hod of recovering high dynamic range fadiance 
maps from photographs taken wiUs conventionaJ imaging equip- 
ment. In our method, multiple photographs of the scene are taken 
with different amtMmts of exposure. Our algorithm uses these dif- 
ferently e?cposed photographs to recover the response ftjnction of the 
imaging proceaa* up to fkc:tor of scale» using the asBiunption of reci^ 
procity. With the known response ftmction» the algtritfcim can fuse 
the multiple photographs into a sin^e, high dynamic range fadience 
map whose pixel values are proportional to the tiue radiance values 
in the scene. We denjonsirate our method on images ac<iuired with 
both photochemical and digital imaging processes. We discuss how 
this work is applicable in many areas of computer graphics involve 
ing digiti^d photographs, including image-based modeling, in^gc 
composittng, and image proc^sing. Lastly, we demosDStrato a few 
applications of having high dynamic range radiance maps, such as 
synthesizing realistic motion bltir and simulating the response of the 
human visual system, 

CR Descriptors: t.2.10 [Ardflrfal Intdligence]: Vision and 
Scene Understanding - Intensity^ coIok photometry and threshold* 
ing; 12.7 [Compater Graphics]: Three^Oimensional Graphics and 
Realism - Color, shading, shadowing^ ond texture; 1.4.1 [Image 
Processing]: Digitization - Scannittg; 1.4.8 [Image Processing]: 
Scene Analysis - Photometry, Sensor Fusion. 

1 Introduction 

Digitized photographs are becoming increasingly important in com- 
puter graphics. More than ever, scanned images are used as texture 
maps for geometric models, and recent work in in^ge-based mod- 
eling and rendering uses images as the fiindamental modeling prim- 
itive. Furthermore, many of today's graphics applications require 
computer-generated images to mesh seamlessly with real photo- 
graphic imagery. Properly using photographically acquired imagery 
in dhese applications can greatly benefit from an accurate model of 
the photographic process. 

When we photograph a scene, either with jflbn or an elec- 
tronic imaging array, and digitize the photograph to obtain a two- 
dimensional array of "brightness'* vahies, these values are rarely 
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tme measurements of relative ladiaiice fn the scene. Por example, if 
one pixel has twice the vahie of another, it is unlikely that it observed 
twice the radiance, tnstead, there is usually an unknown, nonlinear 
mapping that determines how radiance in the scene becomes pixel 
values in the image, 

this nonlinear mapping is hard to know beforehand because it is 
actually the composttton of aevcral nonlin^ mappings that occur 
in the photographic proooss. In a conventional camera (see Fig. J \ 
the film is first exposed to ligbt to form a latem hnage. The film is 
then developed to change this latent image into variations in trans- 
parency* or dmfty, on die film. The film can then be digitized using 
a film scanner, which prcyects ti^ thimigh the fihn onto an elec- 
tronic light-sensitive array, convcrtiiig the image to eJeetrical volt- 
ages. These volta^ ate digitized, and then manipulated before fi- 
nally being written to the storage medium. If prints of the film are 
scanned rather than the film itself^ then the prindng process can also 
introduce nonlinear mappmgs. 

In the first stage of the process, the film reaponSe to variations 
in exposum X (which is the product of theirradiance E the 
film receives and the exposure time Ai) la a noiFb'near function, 
called the "characteristic curve" of the film. Notewortliy in the typ- 
ical characteristic curve is the presence of a small response with no 
exposure and saturation at high exposures, the development, scan- 
ning and digitization processes usually introduce fheir own nonlin- 
earities which compose to give the aggregate nonlinear lelation&hip 
between the image pixel exposures X and their values 

Digital cameras, which use charge coupled device (CCD) arrays 
to image the scene, are prone to the same difficulties; Although the 
charge collected by a CCD element is proportional to its irmdianoe, 
most digital cameras apply a nonlinear mapping to the CCD outputs 
befbie they are written to the storage medhLm* This nonlinear map* 
ping is used in various ways to mimic the response characteristics of 
film, antioipate nonlinear responses in the display device, and often 
to convert 12-bit output fh>m the CCD's analog-to-digital convcirt- 
ers to 8-bit vahies commonly used to store images. As widi fllm^ 
the most significant nonllneadty in the response cmve is at its sat- 
uration point, where any pixel with a radiance above a certain level 
is mapped to the same maximum image vahrn. 

wiry is diis any problem at all? The most obvious difficulty, 
aa any amateur or professional photogmpher knows, is that of lim- 
ited dynamic range — one has to chooae the range of radiance values 
that are of interest and determine the exposure time suitably. Sunlit 
scenes, and scenes with shiny materials and artificial light sources, 
often have extreme differences in radiance values that are impossi- 
ble to capture without either underwexposing or saturating the film. 
To cover the full dynamic range in such a scene, one can take a series 
of photographs with different exposures. This dien poses a prob- 
lem: how can we combine these separate images Into a composite 
radiance map? Here the f&ct that the mapping ftt>m scene radiance 
to pixel values is unknown and rumlinear begins to haunt us. The 
purpose of this paper is to present a simple technique for tecover- 
ing this response fiinctlon, up to a scale factor, using nothing more 
than a set of photographs taken with varying, known exposure du- 
mtions. With this mapping, we then use the pixel values ftom all 
available photographs to construct an accurate map of the radiance 
in the scene, up to a factor of scale. This radiance map will cover 
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Figure ] : Im&ge Acquisition PipeKne sht>ws how scene radiance becomes pixel values for both film and itigitat cameras. Unknown nonlin- 
ear mappings can occur during eiposuJ% devdopmentt scanning, digitizaUotu md remapping, Tfte a^orithm in this paper determines tk& 
^iggregaie mapping Jrom scene radiance L to pixel vahies Zfrom a set qf differently exposed images. 



the entire dynamic range captured hy tine origtat photographs. 
1.1 Applications 

Our technique of derivmg imaging response (bnctlons and recover- 
ing hiith dyxiamie tm^ radifiixift tnxtpB \m tmny possible applicft- 
tions in computer graphics : 

Image-based modeling and renderlns 

tmage-based modeling and rendering systems to date (e.g. [1 1 , 15| 
2, 3, 12, 6> 17]) mak« the Bsmimption that an the images arc taken 
with the same exposure settings and film response ftwctions. How^ 
ever, almost any large-scale environment will have some areas that 
are much brighter than others, making it impossible to adequately 
photogtaph the seene tiling & single ejcposure aettiixg. In indoor 
scenes with windows, this situation often arises within the field of 
view of 9 Single photograph, since the areas visible through the win- 
dows can be far brighter than the areas inside the building. 

By determining the response functions of the imaging device, the 
method presented hete allows one to correctly fvse pixel data ftom 
photographs taken at dijfferem exposui* settings, M a result, one 
can properly photograph outdoor areas with short exposures^ and in- 
door areas with longer exi>osures> without creating inconsistencies 
in the data set Furthermore, knowing the response Auictions can 
be helpful in merging photographs taken with difibrent imaging sys- 
terns, such as video cameras, digital cameras, and film camera^ with 
various 61m stocks and digitization processes- 

The area of Image-based modeling and rendering is working to- 
ward recovering more advanced reflection nxxlels (up to complete 
BRDF's) Of the suffices in the scene (e.g, [21]). These nseth* 
ods, which invDlve observing surface radiance in various directions 
under various lighting conditions, require absolute radiance values 
rather than the nonlinearly mapped pixel values fbund in conven* 
tional images. Just as important, the recovery of high dynamic range 
images will allow these methods to obtain accurate radiance val- 
ues from suri^ce specularitiea and ftt>rt\ incident light sourcea. Such 
higher radiance values usually become clamped in conventional im- 



Image processing 

Most itnage processing operatiom, such as blurring, edge detection* 
color correction, and image correspondence, eixpect pixel values to 
be proportional to the scene radiance. Because of nonliJiear image 
response, especially at the point of saturation, these operations can 
produce incorrect results for convcntiona) itnages. 

In computer graphics, one common image processing operation 
is the application of synthetic motion blur to images. In our re- 
sults (Section 3), we will show that using true radiance uv^ pro- 
duces significantly more realistic motion blur efTe^s for high dy- 
namic range scenes. 



Image compositing 

Many applications in computer graphics involve compositing im- 
age data from images obtained by difTbjfent proceasea. For exam- 
ple, a background matte might be shot widi a sttU camera, Uve 
action mfgiht be shot with a dilTerent fihn stock or scanning pro- 
COBB, and CO elements would be produced by rendering algorithms. 
When there are significant dfcBftwnces hi the response curves of 
these imagmg processea, tbe composite image can be vj$ua]Iy ur^ 
coiwincing, TlKtechniquepreseiiited in^paperprovides aconv&- 
nieni and robust method of deteiroinrng the overall response curve 
of any imaging process, allowing images ftom different processes to 
be used consistently as radiance maps. Furthermore, the recovered 
response curves can be inverted to render the coni^stte radiance 
map as if it had been photographed with any of the original imaging 
processes, or a different imaging process entirely. 

A research tool 

One goal of computer graphics is to simulate the image formation 
process in a way that produces results that are consistent with what 
happens in the real world. Recovering radiance maps of real-wortd 
scenes should alloiw more quantitative evaluations of rendering al- 
gorithms to be made in addition to the qualitative scrutiny they tra- 
ditionally receive, fn particular, the method should be useful for de- 
veloping reflectance and ilhuninatlon models, and comparing global 
ilMminatioD solutions against ground truth data. 

Rendering high dynamic range scenes on Conventional display 
devices is the subject of considerable previous woric, including [20, 
16, 5, 23], The work presented in ttiis paper will allow such meth- 
ods to be tested on real radiance maps in addition to synthetically 
computed radiance sotutionB. 

1.2 Background 

The photochemical processes involved in silver halide photography 
have been the subject of continued innovation and research ever 
since the invention of the daguenetype in 1839. [18] and [8] pro- 
vide a comprehensive treatment of the dieoty and mechanisms in- 
volved, For*fi t)ewer technology of soHd-state injaging with charge 
coupled devices, [19] is an excellent reference. The technical and 
artistic problem of representmg the dynamic range of a natural scene 
on the limited range of film has concerned photographers from the 
early days - [1] presents one of the best known systems to choose 
shutter speeds^ lens apertures» and developing condldona to be^t co- 
erce the dynamic range of a scene to fit into what is possible on a 
print. In scientific applications of photographyt such a^ in astron- 
omy, the nonlinear film response has been addressed by suitable cal- 
ibration procedures. It is Our obiectrvc instead to develop a sirnple 
self-calibrating procedure not requiring calibration charts or photO" 
n^lc measuring devices. 

In previous work, [13] used mwJtiple flux mtegration times of a 
CCD array to acquire extended dynamic range images. Since direct 
CCD outputs were available, the woric did not need to deal with the 
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problem of nonltnear pixel vslue response. tl4] addressed the prob- 
lem of nonlinear response but provide a rather limited method of re- 
covering the response curve. Specifically, a parametric fofm of the 
response curve is arbitrarily assumed, there is no satfsfiEictoty treat- 
ment of image noi$e, and 6ie recovery process maloea only partial 
use of the available data. 

2 The Algorithm 

This section presents our algorithm for recoveritug the film rraponse 
fHinction, and then presents our method of reconstructing the high 
dynamic range radiance image from the multiple photographs. We 
describe the algorithm assuming a grayscale imaging device. We 
discuss how to deal with color in Section 2.6. 

2.1 Film Response Recovery 

Our algorithm is based on exploiting a physical property of tmaging 
systems, both photochemical and electronic^ known as twiprt>ii(iy. 

Let us consider photographic film first. The response of a film 
to variations in exposure is sumuwfl^ed by the characteristic curve 
(or Hurtcr-Driffleld curve). This is a graph of the optical density 
D of the processed fUm against the logarithm of the exposure X 
to which it has been subjected. The eicposure X is deihted a$ the 
product of the irradiance E at the JfUm and exposure time, A*, so 
that its units are Jm"*. Key to the very concept of the characteris- 
tic curve Is the assumption that only the product EAt is rmportant, 
that halving B and doubling At will not change the resulting optical 
density D. Under extreme conditions (very large or very low At )» 
the reciprocity aasurnption can br^ down, a situation described as 
reciprocity failure. In typical print elms, reciprocity holds to within 
^ stop' for exposure times of 10 seconds to 1/10,000 of a second.* 
In d^c case of charge coupled arrays* reciprocity holds under the as- 
sumption that each site measures the total number of photons it ab- 
sorbs during the integration time. 

After the development^ scanning and digitization proce&ses, we 
obtain a digital number ^, which ia a nonlinear ftuiction of the orig- 
inal exposure X at the pixel. Let us call this f\mction /, which is the 
composition of the chamctftristic curve of the film as well as all the 
non linearities introduced by the later processing steps. Our first goal 
will be to recover this fimction /, Once we have that, we can com- 
pute the exposure X at each pist&l, as AT ^ /"^ (Z), We make the 
reasonable assumption that the function / is monotonicaUy tncreas- 
jng^ ?o its inverse /" ^ is well defined. Knowing the exposure X and 
the exposure time At, the iiradiance E h recoveted as ^ = X/At, 
which we will take to be proportional to the radiance in the scene.* 

Before proceeding fbrther, we should discuss the consequences 
of the spectral response of the sensor. The exposure X Should be 
thought of as a function of wavelength X( A), and the abscissa on the 
characteristic curve should be the integral f X{\)R(X]dX where 
is the spectral rejiponse of the sensing element at the pixel lo- 
cation. Strictly speaking, our use of irradiance, a radiometric <iuan- 
tity, is not justified. However, the spectral response ofthc sensor site 
may not be the photopic himinosity junction Va, so the photomet- 
ric tcmi illuminance, is not justifled either. In what follows, we will 
use the term irradiance* while urging the reader to remember that the 



' I Rtop ia a photograptjtc term for a factor of two; ^ Stop is thus 

^An cvan lat*8W dynflmiC range can be covered by iisirtg neutral dcraity 

fitters to lessen to amount of light reaching the film for a given exposure time. 

A discussion of ^ modes of rceiprocity faihiic may be fbond in [IS], ch, d, 
proporHonftt E fbr any partierutar pixd, but it is pOMibIc for the 

proportionality ftrfor (o b<s diflF&fftm diffisront places wl tho acnsor. One 

formula for this vari&nocv given in [7], \aB <= con^f^t whoro or 

mcnsurM tho pixel's atiglc from the lens' optical axis. However, raoat mod- 
orn «AmerA lenses 4ra ocdigned to oompcosatc for this effect, and provide a 
toft^Ot mapping between radiance aiid inEdianoe at f/8 and smaller 
apertures. See also flOj. 



<iuantities we will be dealing with are weighted by the spectral re- 
sponse at the sensor site;. For color photography, the cqIot channels 
may be treated separately. 

the fnpm to our algorithm is a mmiber of digitized photograph» 
taken from the same vantage point with different lcm>wn exposure 
durations A^j.^ We will assume that the scene is static and that this 
process is conopleted quieJdy eiKUgh that lighting changes can be 
safely ignore. It can then be assumed ti)at the fibn irradiance values 
Ei fiir each pixel i are oonstanL We will denote pixel values by 
where % is a spatial indcix over pixels and j indexes over e^^posurc 
times Aty. We may now write down the film reciprocity equarion 
aSr 

Zii^fiBiAi^) (I) 

Since we assume / is monotomc, it is hjveriible, and we can rewrite 
(l)as: 

TaJdng the natural logarithm of both aides, we have: 
ln/~^(^«) = ln£f* + lnA<i 

To simplify notation, let ua define function p =; Iti We then 
have the set of equations: 

Q{Zii)^\aBi-\-\nAi, (2) 

where i ranges over pixels and / ranges over exposure durations. In 
this set of equations, the Z^^ are known, as are the At; . The un- 
knowns are the irradiances Ei^ as wall as the f\inetion ^, although 
we assume that g is smooth and monotonic 

We wish to recover the function g and the inadiances Bi that best 
satisfy the set of equations arising flttfn Bqtjfttjon 2 in a least-squared 
eiTor sense. We note that recovering $ only requires recovering the 
finite number of values that can take since the domain of Z, 
pixel brightness values, is finite. Letting Zmin and Zmam be the 
least and greatest pixel values (integers), TV^ be the number of pixel 
locations and P be the number of photographs, we fbrmulate the 
problem as one of finding the (^moa — ^m<n + 1) values of g{Z) 
and the N values of In Ek that minimize the fbnowh>g quadratic ob- 
jective function; 



(3) 

The first term ensures that the solution satisfies the set of equa- 
tions arising from Equation 2 in a least squares sense. The second 
term is a smoothness term on the sum of squared values of the sec- 
ond derivative of g to ensure that the function g is smooth; in this 
discrete setting we use g"{ii) = - 1) - 2p(z) -1- ^(^ + 1). This 
smoothness term ts essential to the formulation in that jt provides 
coupling between the values g{z) in the minimization. The scalar 
\ weights the smoothness term relative to the data fitting terra, and 
should be chosen appropriately jfor the amount of noise expected in 
the Zy measurements. 

Because it Is quadratic in the ^,-'s and g{^)% minimizing O is 
a straightfbrward linear least squares problem. The overdeterrnincd 

^Most modem SLR cameras have dcctronieally conhroHod ShtJtlm 
which gtvc cxtremoly flceureto Bfid rcproducibla o^posura time*. We tested 
our Cmre\ EOS Elsn camera by ^log a MadntOMi to make digital audio 
recordings of tho ^huttor. By analyzing the»o recordings wc were able to 
verify ihfl accuracy of *he exposuro times to within a ihoosandth of a sec- 
ond. Convcmicntlyt wc determined that ttio actual exposure times varied by 

powers 0 r two between stops ( ^ . jn? * * ^ • i * i * ' « ^* ^» ^> ^ 
than tho rounded number^ displayed ot) the o^mera readout j ^ • ^ < ^< 
J, t, 2,4, 5, 13, 30), Becavso of protfcstns associated with vjgnctttng, 
varying the apcituro is not reoommcndBd. 
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$yBitm of tinoar equations i$ robddtly $olvccl using the singular value 
decomposition (S VD) method An intuitive explanation of the pro* 
ccdure mey be found in Fig. 2. 

Wc need to moke three additional points to complete our descrip" 
tion of the algorithm; 

Flr$t, the solution for the g{x) and Ef values can only be up to 
a single scale factor q. If each log tmdtance value hj Ei were re- 
placed by In Bi + ot, and the function g replaced by ^ + o, the sys- 
tem of equaticmd 2 and also the objective function O would remain 
unchanged. To establish a scale &ctor, we introduce the additional 
constraint giZmut) - 0, where = J(^m»7i + 2ma»), simply 
by adding this as an equation in the linear system. The meaning of 
this constraint is that a pixel with value midway between Zmin and 
Zmam will be assumed to have unit exposure. 

Second, the solution can be made to have a much better fit by an- 
ticinating the basic shape of the response function. Since g{z} will 
typically have a steep slope near Zrwrv wd we should ex- 
pect that g{x) will be less smooth and will fit the data mote poorly 
near these extremes. To recognize tht«, we can ititfoduee a weight- 
ing function w(z) to emphasize the smoothness and fitting terms to* 
ward the middle of the curve. A sensible clioice of to Is a simple hat 
function: 



Equation 3 now becomes: 



Finally^ we need not u$e every available pixel site in this solu- 
tion procedure. Oiven measurements of N pixels in P photographs, 
we have to solve for N values of bijPi aind (^ma» — ^min) sam- 
ples of g. To ensure a sufficiently overdetermined systemi we want 

N(P - 1) > (2max - Zmin)- PoTths piXel ValuO range (Zmam - 

Zmin) = 265, F - 11 photographs, a choice of N on the or- 
der of 50 pixels is more thafl adequate. Sioce the si^e of the sys- 
tem of linear equations arising from Equation 3 is on the order of 
computational complexity considera- 
tions make it impractical to use every pixel location in this algo- 
rithm. Clearly, the pixel locations should be chosen so that they have 
a reasonably even distributjon of pwel values from ^min to Zmaa» 
and so that they are spatially welt distributed fn the image. Further- 
more, the pixels are best sampled from regions of the image with 
low intensity variance so that radiance can be assumed to be con^ 
Slant across the area of the pixel, and the effect of optical blur of tlic 
imaging system is minimized. So far we have perfbrmed this task 
by hand, tJwugh it could easily be automated. 

Note that we have not explicitly cnfbrced the constraint that g 
must be a monotonie flmction. If desired^ this can be done by trans- 
forming the problem to a non-negative least squares problem. Wc 
have not found it necessary because, in our experience, the smooth- 
ness penalty tenn is enough to make the estimated g monotonic in 
addition to being smooth. 

To sliow its simplicity, the matlab routine we used to minimize 
Equation ? is included in the Appendix. Running times are on the 
order of a fiew seconds. 



2.2 Constructing the High Dynamic Range Radi- 
ance Map 

Once the response curve g is recovered, tt can be used to quickly 
conven pixel values to relative mdidnce vahies, assuming the expo- 
sure is known. Note thai the curve can be used to detennine ra- 
diance vahies hi any inutge(s) eoquited by the imaging process asso* 
elated with g, not just the images used to recover the response func- 
tion- 

From Equation 2, we obtain: 

\nEi =g(Zu)- hi Ai^ (5) 

For robustnesSi and to recover high dynamic range radiance val- 
ues, we should use all the available exposures for a particular pixel 
to compute its radiance. For tWs, we reuse the weighting f\inction in 
Equation 4 to give higher weight to exposures in which the pixePs 
value is closer to the middle of the response function; 



Combining the multiple exposures has the effect of reducing 
noise in the Tecoveted radiance values. It also reduces the effects 
of imaging artifacts such as film grain. Sii\ce the weightmg func- 
tion ignores saturated pUel values, 'Tslootning" artifact? have little 
impact on the reconsCiucted radiance values. 

2.2.1 Stordge 

In our implementation the recovered radiance msp is computed as 
an array of single-precision floating point values. For efEJciency, the 
map can be converted to the image format used in the RADIANCE 
[22] simulation and rendering flystetn, which tises just eight bits for 
each of the mantissa and exponent. This format is particularly com- 
pact for color radiance maps. Since it stores just one exponent value 
fbf all three color valties at each pixel. 'Thus, in this format, a high 
dynamic range radiance map requires just one third more storage 
than a conventions! RGB image. 

2<3 How many imagdft are necessary? 

To decide on the number of images needed for the technique, it is 
convenient to consider the two aspects of the process: 

1 . Recovering the film response curve: This requires a minimum 
of two photographs. Whether two photographs are enough 
can be understood in terms of tiDe heuristic explanation of the 
process of filtn response curve recovery shown in Fig. 2. 
if the scene has sufllciently many different radiance values, 
the entire curve can, in principle, be assembled by sliding to- 
gether the sampled curve segments, each with only two sam- 
ples. Note that the photos must be similar enough in their ex- 
posure amounts that some pixels fill) into the working range^ 
of the film in both images; otherwise, there is no infomiarion 
to relQte the exposures to each other Obviously, using morr 
than two images with difTering exposure times improves per^ 
formance with respect to noise sensitivity. 

2. RsCGVering a reliance m<lp given the film re^pQn^e curve* The 
number of photographs needed here is a function of the dy- 
namic range of radiance values in the scene. Suppose the 
rsnge of maximum to minimum radiance values that we are 

^BIoDiTiing occura wticn chaigc er light at highly Sdlur«it&d sites on the 
imaging surface spills over end sff^ values at nclehborlng sites. 

"The worMttgnmge of the film conusponds to the middk aection of the 
raiponnc curve. Th9 ends of tbo curve, in which largo changos in oxpasurc 
cause only small changes in density (or pfxct value), are called the toe and 
the shouiacr. 
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plol of g(Z9) fncA thmft ptnt> obvwved In Itve ImaQsa. attuntno tfiil radl»rK*«t «Bdi pixel 




1flft 405 »0 300 '0 50 100 IM 200 

Figure 2: /fi the figure on the tefi, the x symbob repmem samples of the g curve derived fivm the dts^fat vafues at one pixel far 5 different 
knotvn exposures using Bquatim X The imkaown log irradiance \xiEihQS been arhitranfy assumed tobeO, Nate that the shape of the g curve 
is correcu though its position on the vertical scale is arbitrary corresponding to the unlmown In Ei, 7%e + and o sytv^ta show samples of 
g curve segments derived by consideration of two other pixels; again the vertical position of each segment is arbOram Easentiaify, what we 
want to achieve in the opHmitoiion process is to slide the 3 sampled curve segments up and down (by adfusting their En Ei i) unHi they "line 
up " into o sin^te snipoth, monotonic curve, as shown in the ri&tt figure. The vertical position of the composite curve will remain arbitrary. 



ititere^ted m jrccovcring accurately is R, and the fUm U capa- 
ble of representing in its wording mnfic a dynflanic ««ige of JF. 
Then the mininiiim number of pJiotographs needed is f |i] to 
ensure that every part of the acene is imaged in at least one 
photograph at an eicposure diir^on that puts it in the work- 
ing range of the film response curve. Ae in recovering the re- 
sponse Curve, using more photographs ttian strictly necessary 
wiU reault in better noise sensitivity. 

If one wanted to use as <few photogfsp^hs as poSRiWe, one might 
first recover the response curve of the imaging process by pho* 
tographtng a scene containing a diverse rangp of radiance values at 
three or four different exposureSi diffln-lng by pej^ps one or two 
stops. This response curve could be used to dfitermine the working 
rsnge of the Imagins process, which for the processes we have seen 
would be as many as five tsr six stops. For the remainder of the shoot, 
the photographer could decide jRw any particular scene the number 
of shots necessary to cover its entite dynamic range, For dij^«e in- 
door scenes, only one exposure might be necessary; for scenes with 
high dynamic range, several would be necessary. By recording the 
exposure amount for each shot, the images could then be converted 
to radiance maps using the pre-computed response curve. 

2.4 Racovering dxtendod dynamic range from sin- 
gle Axpoftur«d 

Most commericially available film scanner$ can detect reasonably 
close to the fiiU range of useful densities present in film. However, 
many of these scanners (as well as the Kodak PhotoCD process) pro- 
duce 8-bit-per-channel in^ages designed to be viewed on a sctwn or 
printed on paper. Print film, however, records a signif^cantly grester 
dynamic range than can be displayed with either of these media. As 
a result, such scanners deliver only a portion of the detected dynamic 
range of print film in a single scan, discarding infomiadon in either 
high or low density regions. The portion of the detected dynainic 
range that is delivered can usually be influenced by ••brightness" or 
''density adjustment** controls. 

The method presented tn this paper enables two methods for re- 
covering the fill I dynamic range of print film which wc wilt briefly 



outline^ In the first method^ the print tiegative* is scanned with the 
Scanner set to scan slide film. Most scannera wUl then reconi the 
entire detectable dynamic range of the film in the resulting image. 
As before, a series of differently exposed images of the same scene 
can be used to recover die response function of the imaging system 
with each of these scanner settings. This response function can then 
be used to convert individual exposures to radiance maps. Unfortu- 
nately, since the resulting image is still 8-bitSi)er^hannel, this re- 
sults In increased quantization. 

In the second method, the film can be scanned twice with the 
scanner set to diflbrent density acUustmeitt settings. A series of dif- 
ferently exposed inuigefl of the same scei^ can then t)e used to re- 
cover the response jfunction of the imaging system at each of these 
density acljustment settings. These two response functions can then 
be usil to combine two scans of any single negative using a similar 
technique as in Section 2X 

2.5 Obtaining Absolute Radlanca 

For many applications, such as imago processing and image com- 
positing, the relative radiance values oonpxted by our method are 
alt that arc necessaiy. If needed, an approximation to the scaling 
term necessary to convert to absoltite radiance can be derived using 
the ASA of the film^ and die shutter speeds and exposure amounts tn 
the photographs. With these Ttumbers, formulas that give an approx- 
imate prediction of film response can be fbund in [9]. Such an ap- 
proximation can be adequate for simulatiag visual artifacts such as 
glare, and predicting areas of scott^'c retina) response. If desired^ 
one could recover the sealing ftotor precisely by photographing a 
calibration huninaire of known radiance, and scaling the radiance 
values to agree with the known radiance of the luminaire. 

2.6 Color 

Color images, consisting of red, green* and blue channels, can be 
processed by reconstructing the imaging system respojvse curve for 



'This work was done In coUahowtioo wld> Orogoty w^ I-wwn 
*ConwsttJettt>y» most digital oftmeras also specify their Renshivity in tcmis 
Of ASA. 
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Figure 6: Sixteen pkotogn^hs of a ishurek taken at I^top ineremmisfrm SOsecto see. is directly behind the Hghimost stained 

glass window, making it especiaify bright The blue borders seen in some qfihe image margfna are induced by the image KgfstraHan process. 




Rid (diriwdk Own and ftV* (4«h-4bR>A AM 




(c) (d) 

Figure 7: Recovered response curves for the imaging system used in the church photographs in Fig. 8. (fl-c> Jiesponse functions for the ted, 
green, and blue channels^ platted with the underlying {EiM^yZi^) data shown as light circles, (d) The response functions for red, green, 
and blue plotted on the same axes. Note that while the red and green curves are very consistent, the blue curve rises significantfy above the 
others for low ensure values. This indicates that dark regions in the images exhibU a slight blue cast. Since this arHfact is recovered by the 
response curves, it does not affect the relative radiance values. 
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adjust the bnghtnesa and contrast of the images"* to guarantee that 
each image would be digitized using the same respcnsfi fuji^on. 

Ati unfortunate aspect of the PhotoCD proceaa ia that it does not 
scan precisely the same area of each negative relative to the extents 
of the image." To counteract this effect, we $eometricaUy regis- 
tered the imagw to each other using a using normalaed correlation 
(see {4]) to determine, with sub-pixel accuracy, corregpondiiig pix- 
els between pairs of images. 

Fig. 7(a-c) shows the response ftmctions for the red, green, and 
blue channels of the church Sequence recovered jRrom 28 pixel loca- 
tions. Fi^. 7(d) shows the recovered red, green, and blue response 
curves plotted on the same set of axes. From this plot, we can see 
that while the red and green curves are very consistent, the bhie 
curve rises significantly above the othera P&r low exposure value$. 
Thia indicates that daiic regions in the images exhibit a slight blue 
cast. Since this artifact i$ modeled by the response curves, it will 
not ajRfisct the relative radiance values. 

Fig, 8 interprets the recovered high dynamic tange radiance map 
in a variety of ways. Pig. 8(a) is one of the actual photograi^s, 
which lacks detail in its darkier regions at the same time that nuiny 
values within the two rightmost stained glass windows are saturated. 
Figs. 8(b,c) show the radiance map, linearly scaled to the display de- 
vice using two diffferetit scaling fecton. Although one scaling iiic- 
tor is one thousand times the other, there 1$ u^efii) detail in both im- 
sges. Fig, 8(d) is a false-color image showing radiance values for 
a grayscale version of the radiance map; the highest listed radiance 
value 1$ nearly 250,000 times tliat of the lowest. Figs. 8(e,0 show 
two renderings of the radiance tnap using a new tone reproduction 
algorithm [23]. Although the rightmost stained glass window has 
radiance values over a thousand times higher than the darlcer areas 
in the ral^rs, these renderings exhibit detail in both areas. 

Figure 9 demonstrates two applications of the technic)ues pr^ 
sented in this paper: accurate signal processing and virtual photog- 
raphy. The task is to simulate the effects of motion blur caused by 
moving the camera during the exposure. Fig. 9(a) shows the re- 
sults of convolving an actual, tow-dynamic range photograph with 
a 37 X 1 pixel box filter to simulate horizontal motion blur. Fig. 
9(b) shows the results of applying this same filter to the high dy- 
namic range radiance map, and thon sending thia filtered radiance 
map back through the recovered film response functions using the 
same cKposuro time Ai as in the actual photograph. Because we are 
seeing this image through the actual image response curves, the two 
left images are tonally consistent with each other. However, there is 
a large difference between these two images near the bright spots. In 
the photograph, the bright radiance values have been damped to the 
maximum pixel values by the response function. As a result, these 
clamped values blur with lower neighboring values and fail to satu- 
rate the image in the final result, giving a muddy appeajratice. 

In Fig, 9(b), the extremely high pixel values were represented 
properly in the radiance map and thus remtained at values above the 
level of the response function's saturation point within most of the 
blurred region. As a result, the resulting virtual photograph exhibits 
several crisply-defined saturated regions. 

Fig. 9(c) is an actual photograph with real motion blur induced 
by spinning the camera on the tripod d;jring the exposure^ which is 
equal in duration to Fig. 9(a) and the cjtpoaure simulated in Fig. 
9(b). Clearly, in the bright regions, the bhirring efffect is qualita* 
tivftly similar to the synthetic blur in 9(b) but not 9(^). Th© precise 
shape of the real motion blur is curved and was not modeled fbr this 
demonstration. 



^<*Tht!t fcaturo of the PhotoCD |)rocess Is called '^ccnc Balance Adhist- 
mcrtt", Or SBa. 

^ ' This is far \c99 of a probto for wwinatic npplications, in which the film 
sprocket holes arc xisod to expose and acan prEcisely the same area of each 
frame. 




i 

® 



(b) Synthetically blufred radiance map 



(o) Actual bhirred photograph 
Figure 9; (a) Synthetic motion blur applied to one qfthe origi- 
nal di^tized photographs. The tfight vatu69 in the windows am 
clamped bqfor^ the processing, producing mostly unsaturated vat-^ 
vcy in the blurred regiims. (b) Synthetic motion blur applied to 
a recovered hi^k'tfynamic range radiance map, then virtually re- 
photographed through the recovered film response curves. The ra- 
diance values are clamped to the display device after the ptvces^ing, 
altowing pixels to remain saruratea in the window regions, (c> Jieal 
motion blur created by rotating the camera on the tripod during the 
exposttre, which is much more consistent with (b) than fa). 
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4 Conclusion 

Wc Hvt presented a simple, practical, robust and accurate method 
of rfico veritig high dynamic range radiance mapa from ordMaty pho- 
tographs. Oar method mtA the constramt of sensor reciprocity to 
derive the response j(unction and relative radiance values directly 
frond a set of images taken with different ej^kosujres. This work 1^ 
a wide variety of appUcations in the areas of hnage-bmusd modeling 
and rendering, iniBge processing, and image coiv^slting, a few of 
TKrhich we have demonstrated. It is our hope tha* this work will be 
able to help both regearehers and practitioners of computer graphics 
make much tnore effective use of digitized photogTaphs. 
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A Matlab Code 

Here is the matlab code used to solve the linear system that min- 
imizes the objective function O in Equaiian 3. Given a set of ob- 
served pixel vahws in a set of imases with known expoaurts, this 
routine reconstructs the imaging response curve and the radiance 
values for the given pixels- The weighting function ii}(z) is found 
in Equation 4. 
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JIM BLINN'S CORNER 



Compositing, Part I: Theory 




Assodah'ng & pbcdis color with fts opadtyis the basis for a 
compositing function that Is simple, ehgdnt ^nd general. But there 
0re more reasons than mere prettirwss to store pixels Ms way- 



My cunrcotly favorite journAlistic qiiot« comes from 3 mcig- 
fiztne csllfid Morph's Otitpost on the Distal FronxUt, They 
refer to the c^pexation of avoldlofi jaggles as "atitbftliening.** 
Either thii was a typo or thay thought of the jftggiefi as aUeos. 
This got me ihmkpng about ways to get rid of those croatnrB3 
"-the offepiing oi 3D geoonetiy aod jOittt displays. 

One of the most impotunt aijti-aliajiwg tools in coiitipiiter 
graphie^ comes from a generalizaUon o{ the ^mptc act of 
storing a pixel into a ftame huffer. Several people multacte- 
ously discovered the iiaefulneas of thia opcratioiif so h goes by 
several oatnca: mattltigt image compositing^ elph^ bXeztdiog, 
overlaying^ or Urping' it was most completely codified \n a 
paper by Ptorter and Duff*' where ihey call it tb^ *»ovef " oper- 
ator- Xn thi9 coliimn T'm going to show a new way to derive 
Porter and DufTs **avBr" pperaior and describe some imple- 
mciitailoA detafls that I've fovnd iwefid. la a later ctAma HI 
go into some of the subtleties of how tbU operfttor works with 
iateger pixel anthmetic. 

The bask Ided 

The simplest form of compowtinfi goea as fbllows. Say we 
want to overiay a foreground image on some tmdt^round 
image. The foreground image only covers a part of the back" 
ground; pixels inside the foreground shape will completely 
replace the corresponding tutckground pixels, and psxeU 
outajde the shape leive the background pixels intact. 

Cf we want aoii-aliened edges, thouj^h, things are a bit nHiTB 
complicated. Pixels on tba edge of the shape only partially cover 
the baekground pixels. If the shape Is to be propcriy anti- 
etienedi we must blend the £bre$;round color, F. and bfickgront^d 
color, B, according to the fraction a Ibis value itpresenis the 
percentage of the pixel coveted hy eokir F* The standard way to 
calculate this is to fiund the geometric area ecvercd by R This 
implements a simple box filter for antl-aliening, N<ore Accurate 
filters can he used, but 1^1 stick to the box far now. 

Now let's get dowi» to algebra- F and B ere eaeb three- 
elenient vectors representing the red. green, and blue compo'- 
oents of a pixel. Ordinary vector algebra applies. The new 
color to the frame bufter is 

whtch can be more efBcjcntly calculated as 



finrt^-B« + a(F-B^) 

You can Actually use the valtie of a for a variety of thmgs. In 
addition to iu Bnd*aiienlog ftmction, it can rBpteaent transpar^ 
ent objects or establish a global fade amonnt For (his reason^ 
theCB value also goes by various names! coverage amount, 
opacity, or simply alpha. You can also think of it as 1 minus 
the transperency of the pixel Fm going to cetlit opadty for 
now. If it*s 0, tlw new pixe) is transparent and doca not affect 
the frame buffer. If h4 U the jisw pixel is opaque and com- 
pletely replaces the current ftame buffer cohsr* 

Next, suppose that we want to layer another object on top 
of our image. We jutt blend in the new objeet's eolor, which 
ril call O, on top of onr cvrrent background image using its 
opacity p. 



We can keep on plastering stuff on top of our boage until we 
are happy. This is the essence of 2-1/2D rendering, also known 
as the painter^s algorithm or temporal priority. 

For most randenng purposes Tve been able to provide this 
as the only necessary acceding operation into the frame 
buffer. $ut it*s not quite general enough. 

Associativity 

There is another intrignn^ generaHzation here. Both F and 
O have an Opacity, but B doesn't, poes it even mean anything 
to compooitc into a pixel that already has an opacity? Ye«. 
Consider the followir:^ eceoario. Suppose we have the Images 
F and O, bttt haven't yet dedded what to use f6r a back- 
ground Let*s see if we can mMgeF and O Into one image, K. 
that we can store away and later overlay on B to got the same 
result. If we denote the compositing operatikin with the sym- 
bol &^ what we want is 

(B&F)&G = B&(F&0) 

Xn other words we want to nuke coinpaaitiii^ assodt^ve. 

How can we define H F A O to make this work out? We 
want to calculate a new plirel color H and opacity yin terms of 
colon F and O and their own opacities cs and Plug In the 
denmtlon$: 
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0-«(a-a)B + fltF) + PQ==(l-y>B+YH 

Rean-anse the to get 

(1 - a) (1 -p)B+ ( (a (1 - p)F+po ) =(1 -|)B 

Siac6 we want thio to work for arbltraty baekgivMmdA B« 
c«n spUt this into two e^atioxifi by equetung Uie B coeCfideflts 
and tiifr ooii-B coetGcieDts: 

a-a)a-p)-l'-Y 

The fir^t df tho9e ffV^ us 

Y=ai-P"Otp 
The sGoond equation gives us 

H«^(a(l-p)F+pO))/Y 
With A iittle fiddling thja tunu ioto 

H=(l^%)F+(3/v)G 

Thlfl £ive$ us a definition for bow to composite two colors, 
each of which haa ita own 0|>^ty, 

Lct'5 play with this a bit. If we composite G over & totally 
opaque ooloT F, what Is the result? Plug a « 1 into the abovo 
and w& get 

Y=i 

HRa-P)F+pQ 

Id other words, our more generai coinpouting opsirBtlon boil? 
down to the baaic one if we a»umB our background has Its 
own opacity value^ which happeos to ba X. 

Now let*8 try overlaying a completely opaque color O on F> 
Plug in p 1 with an arbitrary a aod we dnoover 

T=l 
H = 0 

independent of a, as we expect. 

Another form of association 

The above deflnition of H is a bit complicated, Fortunatety, 
there Is a better way. One of die key insists In the Porter and 
Dttff paper h that F ahows up in the compositing formula only 
when multiplied by a» and G appears oxJy wban multiplied by 
p. Why not siniply represent the pixel with the colors ahready 
premuItipHed by their opacity? Thw repreaentation Is usuEiIly 
referred to as having the opadty associaied with the co^oj. I1J 
wr5te<fof the time being) an aasoqiated pixel color with a 
tilde over it. We have 



An associated odor is just a regulair color oomposited onto 
hladc-^at is, if you displayed It directly by itself^ yon would 
get the collect enti-aliened image, (b the joke woni out now? 
OK, I*U use the real word again.) Note that if the apecity equate 
1, an asBocaatad color ia the lama OT an unassocbted color. 

Using theee definitlotts Id the general eomposleing fiinetion 
and doing a Mt of algebr&te fiddling we get 

7^(1-P)a4.p 

This ii a bit less arithmetic than our earlier definition, but 
what makes it particularly pretty it that we am now doing 
exBctly the same arithnKitie 00 the opadiy components of a 
pixel as we are doing on the (associated) color components. 
This is simple, elegant, and gaeeral. 

More reasons to associate 

There are more reasons to store assodated pixd colors than 
mere prettiness of the oompoahing fbimala. For ooe thing, 
some intensity calculation algoritbms directly getierate araod- 
ated pixel colors. Additionally, wemnat use assodated colors 
for any filtering or interpolation operations. Let^ see why. 

AiiHafiastng bytubsampiing 

One typical way to do antiaiiasiog is by subsampHng. You 
calculate an image using point samplfaig at, say. four times 
your final resolution in ;r and y» and then downsample to get 
your final result. There are stiU aJiases, but you have pushed 
them up into higher frequenc|e«» 

How does this vork with our scheme here? You can con- 
sider each final pixel as broken into a 4 x 4 grid of aubpixcl 
cells* each containing a color and an opadty flag. Initialize 
theae all to 0. Then, whenever your roQderer wtitea a color to 
a subplxe] cell have it also set the opadty flag to 1, After 
rendering, sum up the 16 opadty flaga within the pixel and 
call the result yv. The net opacity for tbe pbtat ia NUt, Next, 
sum up tbe 14 color cells in tbe pixel. The average color of the 
pixel Is this sum divided by the number of cells colored. 
But the aisociated color ia cVen n^oie aimply calculated as the 
color sum divided by 16, (lum/iV) * (/V716) » sum/16. You can 
then oompofilte this associated color using the calculated 
opacity, ffn^. In other words* the net ttssodated pixel color 
and opadty is the sum of the subpixels divided 1^ 16. 

This works even better if your rtndcmr is scan-lxAc on- 
ented-^hat is, it visits each pixel once in order left to right, 
top to bottom. You donH need individual aubpixel ceSs. Just 
accumulate the color and opacity into a single pixel cell and 
divide by 16, In practice, i implement this with a .scan-line 
buffer of pixel cells of length equal to the output picture. 
During scan-line processing, each hlgh-resolutiofi pixel gener- 
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attd simply adds ite value to call ogmber jr/4. Theo, every 
four 9can linos, I pvrg^ this buffer by dividing its eottteatt by 
16 and composltfaig It with the bftckgrcmiid usb^ the d«&dci- 
«ted comporitiuft fdrmula. Th^n I zsft> the bulSw in prepara- 
tion for the najn tour »caa Un» 

Ths cloud simulation I tued for Saturn's rings' j^erates a 
pixel's brightness as a product of the color of a doad particle 
times the probability of a paitlelo bfiing both premt in the 
pitxfil «nci illuminated. We can now recognize this as an asiod- 
ated color. Hie Saturn doud simulation also generates a 
transparency value based on probabOitles of bloddng parti- 
cles. The compositing operator I described (or the simulation^ 
U just the associated oompowltion operator, but t <UAi*t T«cog- 
nize it as such at ftrsl. Originally, I actually dh^ided the ocrior 
by the opacity before paadog it to an unassodated onmpnsit- 
iag romine. live and Jeam. 

NiHrfng 

Suppose WB want to fitter an image that has opadties at 
each pixel Do we filter the unaasodated colors F (this was 
oiy firvt thought), or do we jSUer the aa^odated co1ot« To 
find out consider the following thought experimont. 

Lct'B downsaitkple a «ean line by a factor of two In the x 
direction by simply averaging successive pnrs of pixeisi Then 
let's overlay the oo an opaque badcground B. We want 
to arrange things so that downsampllng and overlaying ganei^ 
ate the same color aa overlaying and dowosampling. Let's 
CollQw the adventures of a typical pibtel pair F (with cp«dty ct) 
and O (with opacity 0). Note that F and O are aide by side 
hare, not on top of each Other aa in our earlier examples. 

First, tfy overlaying and then downsampting. Overlay 
(F, a) on getting oF -f (1 - a)B. Overlay (O. on B» get- 
ting ctO 4- (A - P)3. ThsBc two pixels ate now opaque. Now 
downsaxnplc by averaging tbesc results. The color wilt be 

2 2 2 

As long as you composite ISrst, it actually doasn^t matter if 
you do it associated or unasaodated 

Next, let*8 do this in the other order: downumphng fim, 
then overlaying. Downsampting the unassodated coloia and 
opacity^ we get 

color i= (F + Q)/2; opadty =* (a + P)/2 

Now overlay this on B using the unassodated color composit- 
ing function to get 



—the wrong answer. 
Now let*s do this with associated colora. Downsample aP 

and^ 

color w (oF + pO)/2 ; opacity -j Ca + j)/2 

Now overlay this on B usbig the associated composltiotg ftmc- 
tiontoget 

—the right answer. 

To reiterate* dowasampling and, infacti o/^filtarlng opere- 
tlons should operate oa arrays of associated pixel colon as 
well as, of course, oo the array of opadty values. 

Infwpatafton 

Here's another example. Suppose we are doing Oouraud 
inteipolfltion across a poly^. Sach vertex has a color, and 
we do the standard interpolsttou of vertex cdots to get the 
cotoni inside the polygon. Now* ^at if the vertlcea have 
opadtles as well? Wo sbnply inteipolate them in a similar 
manner. But should we Imetpolate uuuSfiOdated colors or 
assodated coloyfl? (1*11 bet you can guess.) 

Actually* this might seem a little open to interprctatioxL 
After am Oouraud interpolation is Itaelf an approximatloa of 
a more accurate curved-snrfboe-shadhig function* Who's to 
say what the correct interpoUtlon amount la? Well, consider 
the fbllowingt Interpolation is another fbimof ^terjng. Sup- 
pose we wanted to expand an image two times by Inteqiolat- 
Ing between cadx fwcel pair. We would again IDce this to look 
the same if we Interpolated and then overiaytd on a back- 
ground or if we ovcrlaycd first and InteipolAted second. 

Going back to potygonsi we mi^iht have a scan line with the 
colors (F, a) on one end end (0, 0) on the other. We want the 
inside colors (o look the same when overlayed onto a back- 
ground. We want to interpolate and then ovctlay over B, and 
we want to make this the same as overtoylng and then tnter- 
polating. 

You can do the algebra yoursdi. Does it look familiar? It's 
just the s&me as the filtering «XftinpK leading us to the con- 
dusion that Oomud iuteipolalion should also be done on 
assodated pixel colors. 

Computfir notatlort 

Each pixel has a red, green^ and blue color bnd an opadty 
a Since we like assodated colors so oiuch, we will represent a 
pu^cl by the quadruple: 

((xF^aF,i^aFw-«<») 

This looks suspicioualy like homogeneous coordhiates. Fve 
tried real hard, but for the life of me I can't Egure out any use 
fbr this observatkM. 
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